Information processing apparatus, electronic apparatus, server, information processing program, and information processing method

ABSTRACT

There is provided an image processing apparatus including a display configured to display a captured image and a representative icon, wherein the representative icon indicates a range of a focus area of the displayed image and the range encompasses a center of focus point located at an initial position within the displayed image, and a processor configured to adjust the range of the focus area of the displayed image according to a size of the representative icon.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/052,911 (filed on Aug. 2, 2018), which is a continuation of U.S.patent application Ser. No. 15/633,928 (filed on Jun. 27, 2017 andissued as U.S. Pat. No. 10,070,045 on Sep. 4, 2018), which is acontinuation of U.S. patent application Ser. No. 14/889,283 (filed onNov. 5, 2015 and issued as U.S. Pat. No. 9,749,519 on Aug. 29, 2017),which is a National Stage Patent Application of PCT International PatentApplication No. PCT/JP2014/002404 (filed on May 2, 2014) under 35 U.S.C.§ 371, which claims the benefit of Japanese Priority Patent ApplicationNo. 2013-104384 (filed on May 16, 2013), the entire contents of whichare incorporated herein by reference.

TECHNICAL FIELD

The present technology relates to an information processing apparatusand an electronic apparatus capable of specifying a parameter at a timewhen an image is refocused by using a user interface (UI), and relatesto a server, an information processing program, and an informationprocessing method for using the apparatuses.

BACKGROUND ART

An image pickup technology is disclosed by which, in an image pickupapparatus, a light field between an objective lens and an image pickupelement is recorded, thereby making it possible to freely change a focusposition after an image is taken and develop an image again (performrefocusing) (see, for example, Patent Literature 1).

Further, as a UI for specifying a focus position and a depth of field,which are necessary parameters for refocusing, a UI has been proposedwhich specifies a focus position by touching a subject intended to befocused on a screen and specifies a depth of field depending on a lengthof time of the touching (see, for example, Patent Literature 2).

CITATION LIST Patent Literature

[PTL 1]

-   Japanese Patent No. 4900723

[PTL 2]

-   Japanese Patent Application Laid-open No. 2012-95186 (third    embodiment)

SUMMARY Technical Problem

However, when the length of time of the touching is used in order tospecify the depth of field, only one of a direction in which the depthof field is increased from an initial value and a direction in which thedepth of field is decreased therefrom can be specified, because time isthe amount that just increases while elapsing.

Further, UIs in related art have other various problems and thus are noteasy to use.

In view of the circumstances as described above, it is desirable toprovide an information processing apparatus and an electronic apparatus,which are provided with an easy-to-use UI, and a server, an informationprocessing program, and an information processing method for using theapparatuses.

Solution to Problem

According to an aspect of the present technology, there is provided animage processing apparatus including: a display configured to display acaptured image and a representative icon, wherein the representativeicon indicates a range of a focus area of the displayed image and therange encompasses a center of focus point located at an initial positionwithin the displayed image; and a processor configured to adjust therange of the focus area of the displayed image according to a size ofthe representative icon.

According to another embodiment of the present technology, there isprovided an image processing method including: controlling arepresentative icon to be displayed to indicate a range of a focus areaof a displayed image, wherein the range encompasses a center of focuspoint located at an initial position within the displayed image; andcontrolling an adjustment of the range of the focus area according to asize of the representative icon.

According to another embodiment of the present technology, there isprovided a non-transitory computer-readable medium having embodiedthereon a program, which when executed by a computer causes the computerto perform an image processing method, the method including: controllinga representative icon to be displayed to indicate a range of a focusarea of a displayed image, wherein the range encompasses a center offocus point located at an initial position within the displayed image;and controlling an adjustment of the range of the focus area accordingto a size of the representative icon.

Advantageous Effects of Invention

As described above, according to the present technology, it is possibleto provide the easy-to-use UI.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing the structure of an image pickup apparatusaccording to an embodiment of the present technology.

FIG. 2 is a functional block diagram showing the entire structure of animage processing unit according to an embodiment.

FIG. 3 is a functional block diagram showing the entire structure of adisplay apparatus of a touch panel type according to an embodiment, towhich the image pickup apparatus is provided.

FIG. 4 is a diagram showing the state where a user touches a subjectSUB1 on the touch panel on which an image IMG0 is displayed, and animage IMG1 is displayed.

FIG. 5 is a diagram showing an example of a depth map DM for obtaining adistance d on the basis of a coordinate position touched by the user onthe touch panel.

FIG. 6 is a diagram showing the state in which the user drags a circleCIR0, which is displayed by touching a subject SUB2 with a forefinger,with a thumb with the circle CIR0 touched, to expand the circle CIR0 toa circle CIR1 having a larger radius.

FIG. 7 is a diagram showing the state where the user drags the circleCIR0 with the thumb, which is displayed by touching the subject SUB2with the forefinger, to expand the circle CIR0 to a circle CIR2 having amuch larger radius.

FIG. 8 is a flowchart for explaining the flow of a process for a UI forrefocusing according to an embodiment.

FIG. 9 is a diagram showing an example in which, when the user touches asubject SUB1, the circle CIR0 is drawn, and a distance from the imagepickup apparatus to the subject SUB1 is indicated with a characterstring STR in the vicinity of a touched position P0.

FIG. 10 is a diagram showing an example in which the user touches twopoints P0 and P1 on the touch panel on which the image IMG0 isdisplayed, and a circle the diameter of which corresponds to a distancebetween the two points touched is drawn.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments according to the present technology will bedescribed with reference to the drawings.

(About Structure of Apparatus)

The structure of a display apparatus according to the present technologywill be described. The display apparatus may include an image pickupapparatus that takes a refocus image. First, the structure of the imagepickup apparatus will be described.

(About Structure of Image Pickup Apparatus)

FIG. 1 is a diagram showing the entire structure of an image pickupapparatus 1 according to an embodiment of the present technology. Theimage pickup apparatus 1 takes an image of an image pickup target object20 and performs image processing, thereby outputting image data Dout.The image pickup apparatus 1 is provided with an image pickup lens 11having an aperture stop 10, a micro lens array 12, an image pickupelement 13, an image processing unit 14, an image pickup element driveunit 15, and a control unit 16.

The aperture stop 10 is an optical aperture stop of the image pickuplens 11. An image of the image pickup target object 20, which is asimilar figure of the shape (for example, circular shape) of theaperture stop 10, is formed on the image pickup element 13 for eachmicro lens.

The image pickup lens 11 is a main lens for taking an image of the imagepickup target object 20 and is formed of, for example, a general imagepickup lens used for a video camera, a still camera, and the like.

The micro lens array 12 is constituted of a plurality of micro lensestwo-dimensionally arranged and is disposed on a focus surface (imageforming surface) of the image pickup lens 11. The micro lenses each havea circular, planner shape and is formed of a solid lens, a liquidcrystal lens, a diffractive lens, or the like.

The image pickup element 13 receives light beams from the micro lensarray 12 and obtains picked-up image data D0 (multi-view image)including a plurality of pieces of pixel data, and is disposed on thefocus surface (image forming surface) of the micro lens array 12. Theimage pickup element 13 is formed of a plurality of CCDs (charge coupleddevices) arranged in a matrix pattern or a two-dimensional solid-stateimage sensor such as a CMOS (complementary metal-oxide semiconductor).

The image processing unit 14 performs predetermined image processing forthe picked-up image data D0 obtained by the image pickup element 13,thereby making it possible to generate image (restructured image,refocus image) data Dout set to any focus. At this time, in the imageprocessing unit 14, predetermined distance information (disparity map ordepth map) is extracted from the picked-up image data D0. On the basisof the distance information, the pieces of picked-up image data arerearranged. The detailed structure of the image processing unit 14 willbe described later. The process by the image processing unit 14 may beachieved by causing a computer to execute a program.

The image pickup element drive unit 15 drives the image pickup element13 and controls a light reception operation thereof.

The control unit 16 controls the operations of the image processing unit14 and the image pickup element drive unit 15 and is formed of amicrocomputer or the like.

Here, with reference to FIG. 2, the detailed structure of the imageprocessing unit 14 will be described. FIG. 2 is a functional blockdiagram showing the entire structure of the image processing unit 14according to an embodiment. The image processing unit 14 is constitutedof a defect correction unit 141, a clamp processing unit 142, a distanceinformation extraction unit 143, a refocus coefficient setting unit 149,a rearrangement processing unit 144, a noise reduction unit 145, anoutline emphasis unit 146, a white balance adjustment unit 147, and agamma correction unit 148.

S10 The detect correction unit 141 corrects a defect (defect due to anabnormality of the image pickup element 13 itself) like being blocked upshadows which is in the picked-up image data D0. The clamp processingunit 142 performs setting process (clamp process) of a black level ofeach of the pieces of pixel data with respect to the picked-up imagedata after the defect correction by the defect correction unit 141. Inaddition, for the picked-up image data that has been subjected to theclamp process, a color interpolation process such as a de-mosaic processmay be performed.

The distance information extraction unit 143 extracts predetermineddistance information on the basis of the picked-up image data D0, andincludes a phase difference detection unit (not shown) and a distanceinformation calculation unit (not shown) as follows.

On the basis of the picked-up image data D1 supplied from the clampprocessing unit 142, the phase difference detection unit generates aplurality of parallax images (any viewpoint images at differentviewpoints) which have different parallaxes and detects a phasedifference between at least two parallax images out of the plurality ofparallax images. It should be noted that the parallax image fordetecting the phase difference can be generated by extracting andsynthesizing pieces of pixel data obtained at a pixel P disposed on thesame position between unit images received on the image pickup element13. Therefore, the number of parallax images generated is equal to thenumber of pixels assigned to one micro lens.

On the basis of the phase difference detected by the phase differencedetection unit, the distance information calculation unit calculatesdistance information (distance d to be described later) between theimage pickup lens and a refocus surface on which the focusing is desiredto be performed (focus position is determined), that is, a focusdistance of the image pickup lens on the object side at the time ofrefocusing.

On the basis of the distance information (calculated by the distanceinformation calculation unit) extracted by the distance informationextraction unit 143 and information of a predetermined depth of field,the refocus coefficient setting unit 149 sets a refocus coefficient αused to perform an integral process (refocus operation process) in therearrangement processing unit 144. It should be noted that the picked-upimage data D1 is input to the rearrangement processing unit 144 with therefocus coefficient α.

The rearrangement processing unit 144 uses the refocus coefficient α setby the refocus coefficient setting unit 149 to perform a predeterminedrearrangement process, for example, the refocus operation process usinga method called “light field photography” for the picked-up image dataD1, with the result that picked-up image data D2 (refocus image) isgenerated.

It should be noted that in the refocus image, depths of field areprovided in a forward direction and a backward direction bypredetermined amounts, with a specified focus position sandwichedtherebetween. The depth of field in the forward direction from the focusposition and the depth of field in the backward direction therefrom maybe equal to each other. Alternatively, both of the depths of field maybe distributed at a certain ratio or may be set to any depths.

The noise reduction unit 145 performs a process of reducing a noise (forexample, noise generated when an image is taken in a dark place or aplace where sufficient sensitivity is not obtained) in the picked-upimage data D2 supplied from the rearrangement processing unit 144. Theoutline emphasis unit 146 performs an outline emphasis process ofemphasizing the outline of an image with respect to the picked-up imagedata supplied from the noise reduction unit 145.

With respect to the picked-up image data supplied from the outlineemphasis unit 146, the white balance adjustment unit 147 performs anadjustment process of color balance (white balance adjustment process)due to an influence of an illumination condition, an individual devicedifference of a spectral sensitivity or the like of the image pickupelement 13, a transmission property of a color filter, or the like.

The gamma correction unit 148 performs a predetermined gamma correction(tone or contrast correction) with respect to the picked-up image datasupplied from the white balance adjustment unit 147, thereby generatingthe picked-up image data Dout.

(About Structure of Display Apparatus)

Subsequently, the entire structure of a display apparatus will bedescribed. FIG. 3 is a functional block diagram showing the entirestructure of a display apparatus 2 of a touch panel type according to anembodiment, to which the image pickup apparatus 1 is provided.

The display apparatus 2 can make a focus adjustment at any position in adisplay image and can perform switching from the image displayed(picked-up image data Dout) to an image set on a different focal plane(picked-up image data DRout). The display apparatus 2 is provided withthe image pickup apparatus 1 including the image processing unit 14, atouch panel 150, a distance information calculation unit 151, therefocus coefficient setting unit 149, the rearrangement processing unit144, the noise reduction unit 145, the outline emphasis unit 146, thewhite balance adjustment unit 147, the gamma correction unit 148, and acontrol unit (not shown) that performs overall control for thosefunctional blocks. It should be noted that, for the same imageprocessing as the image pickup apparatus 1, the same symbols are given,and description thereof will be omitted as appropriate.

The touch panel 150 is constituted of a display control unit and aposition detection unit. The display control unit displays arestructured image, which is based on the picked-up image data Doutoutput from the image pickup apparatus 1, on a display panel (displayunit). The position detection unit detects, when an arbitrary positionof the displayed image is specified, the position specified. Forexample, by touching the display panel with a finger, a pen, or thelike, the touched position is detected by a pressure-sensitive orelectro-static manner. As the display panel, a liquid crystal panel, anorganic EL (electro-luminescence) panel, or the like is used.

The distance information calculation unit 151 selects, from a phasedifference disparity map or a depth map DM of parallax images detectedby the distance information extraction unit 143 (phase differencedetection unit) of the image processing unit 14 described above, a phasedifference (disparity) depending on the position (pixel) detected by thetouch panel 150 and calculates a distance d on the basis of thedisparity.

For example, in the case where a focused image (hereinafter, referred toas image IMG1) is desired to be generated on the specified position ofthe displayed image (hereinafter, referred to as image IMG0), thedistance d is set as a distance between the image pickup lens and such arefocus surface that the focus is obtained on the position detected bythe touch panel 150 and is calculated by using the disparity on theposition detected.

In the display apparatus 2 as described above, on the touch panel 150,when the display control unit displays the image IMG0 based on thepicked-up image data Dout and specifies the arbitrary position of theimage IMG0, the specified position is detected, and information relatingto the position (position information) and information relating to thedepth of field (depth-of-field information) are input to the distanceinformation calculation unit 151.

On the other hand, to the distance information calculation unit 151, thedisparity map or the depth map DM of the parallax images are input fromthe position information extraction unit 143 (phase difference detectionunit) in the image processing unit 14 of the image pickup apparatus 1.

In the distance information calculation unit 151, for example, after thedisparity corresponding to the specified position is selected, thedistance d is calculated on the basis of the disparity selected andinput to the refocus coefficient setting unit 149 along with thepicked-up image data Dout.

In the refocus coefficient setting unit 149, as described above, on thebasis of the distance d and the depth-of-field information, the refocuscoefficient α is set and input to the rearrangement processing unit 144along with the picked-up image data Dout.

In the rearrangement processing unit 144, on the basis of the refocuscoefficient α, the rearrangement process for the picked-up image dataDout is performed, and the image (picked-up image data D3) focused onthe specified position in the image IMG0 is restructured, for example.The picked-up image data D3 restructured as described above is subjectedto predetermined image processing by the noise reduction unit 145, theoutline emphasis unit 146, and the white balance adjustment unit 147,and the gamma correction unit 148, and is input to the display controlunit of the touch panel 150 as picked-up image data DRout. In this way,the display control unit displays the image IMG1 focused on thespecified position in the image IMG0 on the display panel.

As described above, the disparity is detected depending on the specifiedposition of the image IMG0, the predetermined distance information iscalculated to set the refocus coefficient α, and the rearrangement ofthe picked-up image data is performed. Therefore, it is possible toinstantly perform switching from the image IMG0 to the image IMG1 thathas been subjected to the focus adjustment at the arbitrary position anddisplay the image.

Modified Example 1 (Separation of Image Pickup Apparatus and DisplayApparatus)

It should be noted that the image pickup apparatus 1 and a displayapparatus 2B may be separately provided, although the display apparatus2 includes the image pickup apparatus 1 in the above description. Inthis case, to the display apparatus 2B, the picked-up image data Doutand the DM are supplied from the image pickup apparatus 1 outsidethereof or a server or the like on a cloud.

Modified Example 2 (Simplification of Display Apparatus and Utilizationof Cloud)

A display apparatus 2C having the structure in which the distanceinformation calculation unit 151, the refocus coefficient setting unit149, the rearrangement processing unit 144, the noise reduction unit145, the outline emphasis unit 146, the white balance adjustment unit147, and the gamma correction unit 148 are removed from the displayapparatus 2B in the modified example 1 may be used. In this case, to thedisplay apparatus 2C, first, the picked-up image data Dout is suppliedfrom the server on the cloud, and the image IMG0 is displayed on thedisplay panel of the touch panel 150.

When a user touches a position of a subject intended to be focused tospecify the position information thereof and specifies thedepth-of-field information by a method to be described later, thedisplay apparatus 2C transmits the position information and thedepth-of-field information to the server.

The server includes the distance information calculation unit 151, therefocus coefficient setting unit 149, the rearrangement processing unit144, the noise reduction unit 145, the outline emphasis unit 146, thewhite balance adjustment unit 147, and the gamma correction unit 148.The server generates the picked-up image data DRout with thosefunctional blocks and transmits the data to the display apparatus 2C.

The display apparatus 2C displays the picked-up image data DRoutsupplied from the server on the display panel of the touch panel 150 asthe image IMG1.

(About UI for Refocusing)

Subsequently, detailed description of an UI for specifying the focusposition and the depth of field at the time of refocusing.

(Specification of Focus Position)

First, a method of specifying the focus position on the UI according tothe present technology will be described. The focus position at the timewhen the user performs refocusing at a desired position is specified bytouching the subject intended to be focused on the image IMG0, which isdisplayed on the touch panel 150, by the user.

FIG. 4 is a diagram showing the state where the user touches a subjectSUB1 on the touch panel 150 on which the image IMG0 is displayed, andthe image IMG1 is displayed. As is apparent from the figure, the usertouches the subject SUB1, thereby specifying the distance d up to thesubject SUB1 as the focus position and performing refocusing to put thesubject SUB1 into focus. As a result, the image IMG1 in which subjectSUB2 and the like at the rear of the subject SUB1 are blurred isdisplayed.

Further, in the vicinity of a touched position P0, a circle CIR0 thatrelatively indicates the depth of field of the image IMG1 is displayed.

It should be noted that the circle CIR0, which has the center at theposition P0 on the touch panel 150 touched by the user, is drawn with aprescribed value or the relative size of the depth of field specifiedlast time as a radius thereof.

(Example of Depth Map)

FIG. 5 shows an example of the depth map DM for obtaining the distance don the basis of a coordinate position touched by the user on the touchpanel 150, although the depth map is not displayed on the display panelas the UI.

In the figure, the distance d of each subject from the image pickupapparatus is indicated in a gray-scale manner. For one subject, anapproximately uniform distance d is obtained, so the one subject isshown so as to have an approximately uniform color density. It should benoted that in an actual depth map DM, pixels have values of thedistances d, respectively.

When the user touches the touch panel 150, the position detection unitdetects XY coordinates of the touched position and checks thecoordinates against the depth map DM. As a result, it is possible toobtain the distance d of the touched position.

The depth map DM can be generated by providing the distance d calculatedon the basis of the disparity map described above as the value for eachpixel.

(Specification of Depth of Field)

Subsequently, a method of specifying the depth of field in the UI of thepresent technology will be described. To perform refocusing at a desireddepth of field by the user, the depth of field is specified by changingthe size of the circle CIR0 drawn around the position P0 initiallytouched by the user to specify the focus position. The changing the sizeis performed by changing a distance between the positions P0 and P1 bydragging the position P1 touched next.

FIG. 6 is a diagram showing the state in which the user drags the circleCIR0, which is displayed by touching the subject SUB2 with a forefinger,with a thumb with the circle CIR0 touched, to expand the circle CIR0 toa circle CIR1 having a larger radius.

In the figure, the depth of field is still small. Therefore, the subjectSUB2 is brought into focus, but the subject SUB1 disposed before thesubject SUB2 is blurred. Further, a subject SUB3 and the like at therear of the subject SUB2 are also blurred.

Subsequently, the state where the depth of field is further changed isshown. FIG. 7 is a diagram showing the state where the user drags thecircle CIR0 with the thumb, which is displayed by touching the subjectSUB2 with the forefinger, to expand the circle CIR0 to a circle CIR2having a much larger radius.

In the figure, the depth of field is further increased. Therefore,unlike FIG. 6, not only the subject SUB2 but also the subjects SUB1 infront thereof and the subject SUB3 at the rear thereof are brought intofocus. The description of the UI for the refocusing is given above.

(About Flow of Process for UI)

Subsequently, a flow of a process for a UI for refocusing according tothe present technology will be described. FIG. 8 is a flowchart forexplaining the flow of the process for the UI for refocusing.

(Process in Case where Two Points are Touched)

A process in the case where a first point is touched, and then a secondpoint is touched will be described first.

First, the image processing unit 14 sets parameters of the focusposition and the depth of field to initial values and uses those initialvalues to generate the picked-up image data Dout. (Step S1)

Then, the display control unit of the touch panel 150 displays the imageIMG0 based on the picked-up image data Dout on the display panel. (StepS2)

Then, the position detection unit of the touch panel 150 determineswhether a first point touch on the touch panel 150 is performed or not.(Step S3)

In the case where the first point touch is performed (Yes in Step S3),the distance information calculation unit 151 obtains coordinates of thetouched position P0 from the position detection unit and determines thedistance d of the position P0 on the basis of the depth map DM obtainedfrom the image processing unit 14. (Step S4)

Then, the refocus coefficient setting unit 149 changes the parameter ofthe focus position from the distance d obtained and resets the refocuscoefficient α. (Step S5)

Then, the rearrangement processing unit 144 uses the reset refocuscoefficient α to restructure the refocus image and generate the imageIMG1. The image IMG1 generated is displayed on the display panel by thedisplay control unit of the touch panel 150. (Step S6)

Then, the display control unit of the touch panel 150 draws a circlehaving a size corresponding to the set depth of field on the displaypanel. (Step S7)

Then, the position detection unit of the touch panel 150 determineswhether a second point touch on the touch panel 150 is performed or not.(Step S8)

In the case where the second point is performed (Yes in Step S8), thedistance information calculation unit 151 obtains coordinates thetouched position P1 from the position detection unit and calculates adistance between the positions P0 and P1 from the coordinates of those.(Step S9)

Then, the refocus coefficient setting unit 149 resets the parameter ofthe depth of field from the distance calculated. (Step S10)

After Step S10, the display apparatus 2 returns the control to Step S4and continues the process.

This is the process in the case where the two points are touched.

(Process in Case where First Point Touch is not Performed)

In Step S3, when the first point touch is not performed (No in Step S3),the position detection unit of the touch panel 150 then determineswhether a certain time period elapses without the first point touch.(Step S11)

In the case where the certain time period does not elapse yet (No inStep S11), the display apparatus 2 returns the control to Step S3.

In the case where the certain time period elapses (Yes in Step S11),then, the refocus coefficient setting unit 149 returns the parameter ofthe depth of field to the initial value and resets the parameter. (StepS12)

After Step S12, the display apparatus 2 returns the control to Step S3.This is the process in the case where the first point touch is notperformed.

(Process in Case where First Point Touch is Performed, but Second PointTouch is not Performed)

In Step S8, in the case where a second point touch is not performed (Noin Step S8), then, the position detection unit of the touch panel 150determines whether the first point touch is maintained or not. (StepS13)

In the case where the first point touch is maintained (Yes in Step S13),the display apparatus 2 returns the control to Step S4 and continues theprocess.

In the case where the first point touch is not maintained (No in StepS13), the display apparatus 2 returns the control to Step S3 andcontinues the process.

This is the process in the case where the first point touch isperformed, but the second point touch is not performed.

Modified Example 3 (Display of Distance)

In the above description, the circle is displayed when the first pointtouch is performed. However, when the circle is drawn, a distance to atouched subject may be displayed at the same time.

FIG. 9 is a diagram showing an example in which, when the user touchesthe subject SUB1, the circle CIR0 is drawn, and a distance from theimage pickup apparatus 1 to the subject SUB1 is indicated with acharacter string STR in the vicinity of the touched position P0.

Modified Example 4 (Circle Diameter of which is Distance Between TwoPoints Touched)

In the above description, the circle, the center of which is the firstpoint touched, and the radius of which is the distance between the firstpoint and the second point touched, is drawn. A circle, a diameter ofwhich is a distance between two points touched, may be drawn. In thisstructure, a position of a midpoint of a line segment that connects thetwo points touched is specified as the position to be focused. It shouldbe noted that the two points may be touched at the same time or may betouched one by one successively.

FIG. 10 is a diagram showing an example in which the user touches thetwo points P0 and P1 on the touch panel 150 on which the image IMG0 isdisplayed, and a circle the diameter of which corresponds to a distancebetween the two points touched is drawn. As shown in the figure, in thecase where the circle the diameter of which is the distance of thetouched two points is drawn, it is unnecessary to put a finger on thecenter of the circle. Therefore, it is advantageous to prevent a part ofthe focus position from being hidden by the finger and make the partmore visible.

Modified Example 5 (Representation of Depth of Field with Graphics)

In the above description, to represent the relative depth of the depthof field, the circle is drawn. However, the way of representation is notlimited to this. For example, a length of a line segment or a size ofanother graphic may represent the relative depth of the depth of field.

Examples of other graphics include a triangle, a square, a heart, ashape that represents a diaphragm plate of a camera, and the like.

Modified Example 6 (Specification of Aperture Value with Circle)

In the above description, the size of the circle represents the relativedepth of the depth of field. However, the size of the circle mayrepresent not the depth of field but an aperture value of a lens of acamera. In this case, when the circle becomes smaller, the diaphragm isnarrowed down, and the depth of field is increased. When the circlebecomes larger, the diaphragm is opened, and the depth of field isdecreased.

Modified Example 7 (Indication of Aperture Value)

In the modified example 3 described above, the distance is indicated inthe vicinity of the position of the first point touched, but an aperturevalue corresponding to a depth of field set may be indicated.

Modified Example 8 (Predetermined Size of Circle)

The size of the circle displayed so as to correspond to an initial valueof the depth of field may be selected to be appropriate in accordancewith the size or resolution of the display panel of the touch panel 150.

For example, the size of the displayed image IMG0 is significantlydifferent between the case where the display apparatus 2 is a televisionset having a screen size of 50 inch and the case where the displayapparatus 2 is a 5-inch smart phone. However, the size of a hand of auser who operates the both is the same, so the size of the circledisplayed as a predetermined value is adjusted so as to be a distance bywhich the user easily touches with fingers, making it easy to use.

For the display apparatus 2 having a 9-inch display panel, it is thoughtthat a circle having a radius of 2 cm is drawn, for example.

Of course, the predetermined size of the circle may be settable by theuser.

Modified Example 9 (Predetermined Way of Blurring)

In the above description, for the depth of field at the time of thefirst point touch, the set value as the initial value is used first, andthe refocus image is restructured. The initial value may be changed bythe user by setting. With this structure, a predetermined way ofblurring at the time of the first point touch by the user can be changedto a desired way of blurring by the user.

Modified Example 10 (Reset Area)

For the UI according to the present technology, a reset area may beprovided. The reset area receives an instruction for explicitlyresetting the focus position and the depth of field to the initialvalues by the user, after the user causes the refocus image to bedisplayed on the display panel on the basis of the arbitrary focusposition and depth of field.

When the user touches the reset area, it is possible to easily reset thefocus position and the depth of field to the initial values.

Modified Example 11 (Maintenance of Depth of Field after Hand isReleased)

In the UI according to the present technology, after the user touchesand operates the touch panel 150 to specify the focus position and thedepth of field, even if the user releases the hand from the touch panel150, the image with the depth of field specified may be displayed for acertain time period.

With this structure, the user can release the finger from the touchpanel 150 and view the refocus image with the display panelunobstructed.

Modified Example 12 (Reset Because of No Operation for Certain TimePeriod)

In the UI according to the present technology, in the case where theuser does not touch the touch panel 150 for a certain time period, thefocus position and the depth of field of the refocus image may be resetto the initial values.

Modified Example 13 (Entire Focus Image in Case where Specified Value ofDepth of Field Exceeds Predetermined Value)

In the UI according to the present technology, in the case where thedepth of field specified by the user exceeds the predetermined value,the display apparatus 2 may generate an entire focus image as therefocus image.

Modified Example 14 (to Generate Refocus Image in Advance)

In the above description, each time the user touches the touch panel150, the refocus image is restructured. However, the structure is notlimited to this. From a multi-view image, a plurality of imagescorresponding to the focus position and the depth of field which may bespecified by the user may be structured in advance and stored in astorage unit provided to the server or the display apparatus.

This structure can save time to restructure the refocus image after theuser touches the touch panel, so it is possible to display the refocusimage at a high speed.

Modified Example 15 (Preparation of Image by Typical Camera)

In the modified example 14, from the multi-view image, the plurality ofimages corresponding to the focus position and the depth of field whichmay be specified by the user is structured in advance. However, thestructure is not limited to this. By using an ordinary camera, aplurality of images corresponding to the focus position and the depth offield which may be specified by the user may be taken and stored in astorage unit provided to the server or the display apparatus.

With this structure, the user can view the refocus image with thedisplay apparatus without using the light field camera such as the imagepickup apparatus 1.

Modified Example 16 (Specification of Position by not Touching)

In the above description, when the user touches the touch panel 150, thefocus position and the depth of field are specified. However, thestructure is not limited to this. The focus position and the depth offield may be specified by a non-contact position detection apparatus orby using a mouse.

Modified Example 17 (Specification of Depth of Field by not TouchingCircle)

In the above description, to specify the depth of field, on the touchpanel 150, the user touches the circumference of the circle displayed bythe first point touch and slides the finger while touching thecircumference, thereby changing the depth of field. However, thestructure is not limited to this. Even if the user does not touch thecircumference of the circle, any position touched on the touch panel 150may be received as a second point touch position.

Also in this case, when the user slides the finger from the positiontouched as the second point, the size of the circle displayed is scaledup or down depending on the distance between the first and second touchpositions.

The present technology may be embodied as the following configurations.

(1) An image processing apparatus including:

-   -   a display configured to display a captured image and a        representative icon, wherein the representative icon indicates a        range of a focus area of the displayed image and the range        encompasses a center of focus point located at an initial        position within the displayed image; and    -   a processor configured to adjust the range of the focus area of        the displayed image according to a size of the representative        icon.        (2) The image processing apparatus of (1), further including:    -   a touch panel configured to receive a re-sizing operation input        from a user, the re-sizing operation being made to change the        size of the representative icon.        (3) The image processing apparatus of (1) or (2), wherein the        focus area is an area of the displayed image that meets a focus        criteria.        (4) The image processing apparatus of any of (1) through (3),        wherein the focus criteria is a condition of being in-focus.        (5) The image processing apparatus of any of (1) through (4),        wherein the display is further configured to re-display the        image with the adjusted range of the focus area.        (6) The image processing apparatus of any of (1) through (5),        wherein the touch panel is further configured to receive a        re-locating operation input from the user, the re-locating        operation being made to move the center of focus point from the        initial position to a relocated position within the displayed        image, and the processor is further configured to recompose the        displayed image by refocusing based on the relocated position of        the center of focus point.        (7) The image processing apparatus of any of (1) through (6),        wherein the center of focus point is moved by dragging and        dropping from the initial position to the relocated position        within the displayed image.        (8) The image processing apparatus of any of (1) through (7),        wherein the processor is further configured to recompose the        displayed image by refocusing based on the relocated position of        the center of focus point.        (9) The image processing apparatus of any of (1) through (8),        wherein the recomposing the displayed image by refocusing        includes resetting the range of the focus area to encompass the        relocated position of the center of focus point as a central        point of the range of the focus area.        (10) The image processing apparatus of any of (1) through (9),        wherein the re-sizing operation is a pinch-to-zoom operation.        (11) The image processing apparatus of any of (1) through (10),        wherein the representative icon includes a circular icon having        boundaries that bound the range of the focus area.        (12) The image processing apparatus of any of (1) through (11),        wherein the representative icon includes a line segment icon        having one end positioned at a far end of the range of the focus        area and another end positioned at a near end of the range of        the focus area.        (13) The image processing apparatus of any of (1) through (12),    -   wherein the representative icon includes a circular icon, and    -   wherein the range of the focus area is decreased when the user        performs the re-sizing operation to increase the size of the        representative icon, and the range of the focus area is        increased when the user performs the re-sizing operation to        decrease the size of the representative icon.        (14) An image processing method including:    -   controlling a representative icon to be displayed to indicate a        range of a focus area of a displayed image, wherein the range        encompasses a center of focus point located at an initial        position within the displayed image; and    -   controlling an adjustment of the range of the focus area        according to a size of the representative icon.        (15) The image processing method of (14), wherein the focus area        is an area of the displayed image that meets a focus criteria.        (16) The image processing method of (14) or (15), wherein the        focus criteria is a condition of being in-focus.        (17) The image processing method of any of (14) through (16),        further including:    -   re-displaying the image with the adjusted range of the focus        area.        (18) The image processing method of any of (14) through (17),        wherein the size of the representative icon is changed according        to a user operation.        (19) The image processing method of any of (14) through (18),        further including:    -   re-displaying the image with the adjusted range of the focus        area based on the changed size of the representative icon.        (20) The image processing method of any of (14) through (19),        wherein the user operation is a pinch-to-zoom operation.        (21) The image processing method of any of (14) through (20),        further including:    -   moving the center of focus point from the initial position to a        relocated position within the displayed image.        (22) The image processing method of any of (14) through (21),        wherein the center of focus point is moved by dragging and        dropping from the initial position to a relocated position        within the displayed image.        (23) The image processing method of any of (14) through (22),        further including:    -   recomposing the displayed image by refocusing based on the        relocated position of the center of focus point.        (24) The image processing method of any of (14) through (23),        wherein the recomposing the displayed image includes resetting        the range of the focus area to encompass the relocated position        of the center of focus point as a central point of the range of        the focus area.        (25) The image processing method of any of (14) through (24),        wherein the representative icon includes a circular icon having        boundaries that bound the range of the focus area.        (26) The image processing method of any of (14) through (25),        wherein the representative icon includes a line segment icon        having one end positioned at a far end of the range of the focus        area and another end positioned at a near end of the range of        the focus area.        (27) The image processing method of any of (14) through (26),    -   wherein the representative icon includes a circular icon, and    -   wherein the range of the focus area is decreased when a user        performs an operation to increase the size of the representative        icon, and the range of the focus area is increased when the user        performs an operation to decrease the size of the representative        icon.        (28) A non-transitory computer-readable medium having embodied        thereon a program, which when executed by a computer causes the        computer to perform an image processing method, the method        including:    -   controlling a representative icon to be displayed to indicate a        range of a focus area of a displayed image, wherein the range        encompasses a center of focus point located at an initial        position within the displayed image; and    -   controlling an adjustment of the range of the focus area        according to a size of the representative icon.        (29) The non-transitory computer-readable medium of (28),        wherein the focus area is an area of the displayed image that        meets a focus criteria.        (30) The non-transitory computer-readable medium of (28) or        (29), wherein the focus criteria is a condition of being        in-focus.        (31) An information processing apparatus, comprising:    -   a control unit configured to        -   cause a first image of a subject taken by an image pickup            apparatus to be displayed on a display screen,        -   receive a first position and a second position selected by a            user on the first image displayed on the display screen, the            second position being different from the first position,        -   calculate a position of the subject to be focused on the            basis of at least one of the first position and the second            position,        -   calculate a depth of field corresponding to a distance            between the first position and the second position, and        -   perform focusing on the calculated position of the subject            and cause a second image having the calculated depth of            field to be displayed on the display screen.            (32) The information processing apparatus according to (31),            wherein    -   the control unit causes a graphic having a size corresponding to        the calculated depth of field to be displayed on the display        screen.        (33) The information processing apparatus according to (32),        wherein    -   the control unit displays, as the graphic, a circle which has a        center on the first position and has a radius corresponding to        the distance between the first position and the second position.        (34) The information processing apparatus according to (33),        further comprising    -   an image processing unit configured to generate the first image        and the second image on the basis of picked-up image data        obtained in a state in which a traveling direction of a light        beam is maintained by an image pickup optical system having a        micro lens array between an image pickup lens and an image        pickup element that obtains the picked-up image data on the        basis of the light beam received, the micro lens array including        micro lenses each of which are assigned to a plurality of pixels        of the image pickup element.        (35) The information processing apparatus according to (34),        wherein    -   the image processing unit includes        -   a distance information calculation unit that calculates            distance information from the image pickup lens to a refocus            surface in accordance with the first position received by            the control unit,        -   a refocus coefficient setting unit that sets a refocus            coefficient on the basis of the depth of field and the            distance information calculated by the distance information            calculation unit, and        -   a rearrangement processing unit that rearranges the            picked-up image data on the basis of the refocus coefficient            set by the refocus coefficient setting unit, thereby            generating an image on any focus.            (36) The information processing apparatus according to (32),            wherein    -   the control unit displays, as the graphic, a circle having a        diameter corresponding to a line segment that connects the first        position with the second position.        (37) An electronic apparatus, comprising:    -   a control unit configured to        -   cause a first image of a subject taken by an image pickup            apparatus to be displayed on a display screen,        -   receive a first position and a second position selected by a            user on the first image displayed on the display screen, the            second position being different from the first position,        -   calculate a position of the subject to be focused on the            basis of at least one of the first position and the second            position,        -   calculate a depth of field corresponding to a distance            between the first position and the second position, and        -   perform focusing on the calculated position of the subject            and cause a second image having the calculated depth of            field to be displayed on the display screen;    -   a display unit including the display screen; and    -   a position detection unit capable of detecting the first        position and the second position.        (38) A server, comprising    -   a control unit configured to        -   transmit a first image of a subject taken by an image pickup            apparatus to a terminal apparatus,        -   receive a first position and a second position selected by a            user on the terminal apparatus on which the first image is            displayed, the second position being different from the            first position,        -   calculate a position of the subject to be focused on the            basis of at least one of the first position and the second            position,        -   calculate a depth of field corresponding to a distance            between the first position and the second position, and        -   perform focusing on the calculated position of the subject            and cause a second image having the calculated depth of            field to be transmitted to the terminal apparatus.            (39) An information processing program causing a computer to            function as a control unit configured to    -   cause a first image of a subject taken by an image pickup        apparatus to be displayed on a display screen,    -   receive a first position and a second position selected by a        user on the first image displayed on the display screen, the        second position being different from the first position,    -   calculate a position of the subject to be focused on the basis        of at least one of the first position and the second position,    -   calculate a depth of field corresponding to a distance between        the first position and the second position, and    -   perform focusing on the calculated position of the subject and        cause a second image having the calculated depth of field to be        displayed on the display screen.        (40) An information processing method, comprising:    -   causing a first image of a subject taken by an image pickup        apparatus to be displayed on a display screen;    -   receiving, by a control unit, a first position and a second        position selected by a user on the first image displayed on the        display screen, the second position being different from the        first position;    -   calculating a position of the subject to be focused on the basis        of at least one of the first position and the second position by        the control unit;    -   calculating a depth of field corresponding to a distance between        the first position and the second position by the control unit;        and    -   performing focusing on the calculated position of the subject        and cause a second image having the calculated depth of field to        be displayed on the display screen.

Supplementary Matter

In addition, the present technology is not limited to the aboveembodiments and can of course be variously changed without departingfrom the gist of the present technology.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

-   1 Image pickup apparatus-   2-2C Display apparatus-   10 Aperture stop-   11 Image pickup lens-   12 Micro lens array-   13 Image pickup element-   14 Image processing unit-   15 Image pickup element drive unit-   16 Control unit-   141 Defect correction unit-   142 Clamp processing unit-   143 Distance information extraction unit-   144 Rearrangement processing unit-   145 Noise reduction unit-   146 Outline emphasis unit-   147 White balance adjustment unit-   148 Gamma correction unit-   149 Refocus coefficient setting unit-   150 Touch panel-   151 Distance information calculation unit

1. An image processing apparatus comprising: at least one processorconfigured to: obtain a first touch input to a first position on atouch-panel display; set a focus position of a captured image on a basisof a distance information of the captured image corresponding to thefirst position; control, on a basis of the focus position, thetouch-panel display to display a refocus image based on a depth of fieldcorresponding to the focus position and display a diaphragm-plate iconat the first position over the refocus image, wherein thediaphragm-plate icon has a shape representing a diaphragm-plate of acamera and indicates an aperture value corresponding to the depth offield; obtain a second touch input to a second position on thetouch-panel display, wherein the second touch input is a drag operationand the second position is different from the first position; and adjustthe depth of field on the basis of the drag operation while changing theaperture value of the diaphragm-plate icon.
 2. The image processingapparatus according to claim 1, wherein the depth of field includes aforward depth of field and a backward depth of field provided in aforward direction and a backward direction respectively from the focusposition, and the at least one processor is configured to simultaneouslyadjust, on the basis of the drag operation, the forward depth of fieldand the backward depth of field while changing the aperture value of thediaphragm-plate icon.
 3. The image processing apparatus according toclaim 2, wherein the at least one processor is configured to:simultaneously increase, on the basis of the drag operation, the forwarddepth of field and the backward depth of field while decreasing theaperture value of the diaphragm-plate icon; and simultaneouslydecreasing, on the basis of the drag operation, the forward depth offield and the backward depth of field while increasing the aperturevalue of the diaphragm-plate icon.
 4. The image processing apparatusaccording to claim 1, wherein the at least one processor is furtherconfigured to determine the distance information corresponding to thefirst position on a basis of the first position and a phase differencebetween a plurality of viewpoint images obtained with a plurality oflenses arranged two-dimensionally.
 5. The image processing apparatusaccording to claim 1, wherein the at least one processor is configuredto control, when obtaining the first touch input, the touch-paneldisplay to initiate display of the diaphragm-plate icon to have anaperture value corresponding to a depth of field specified last time. 6.The image processing apparatus of claim 1, wherein the second positioncorresponds to a circumference of the diaphragm-plate icon.
 7. The imageprocessing apparatus of claim 1, wherein the at least one processor isconfigured to obtain the second touch input after displaying thediaphragm-plate icon, and wherein the second position corresponds to aposition different from a circumference of the diaphragm-plate icon. 8.An image processing method comprising: obtaining a first touch input toa first position on a touch-panel display; setting a focus position of acaptured image on a basis of a distance information of the capturedimage corresponding to the first position; controlling, on a basis ofthe focus position, the touch-panel display to display a refocus imagebased on a depth of field corresponding to the focus position anddisplay a diaphragm-plate icon at the first position over the refocusimage, wherein the diaphragm-plate icon has a shape representing adiaphragm-plate of a camera and indicates an aperture valuecorresponding to the depth of field; obtaining a second touch input to asecond position on the touch-panel display, wherein the second touchinput is a drag operation and the second position is different from thefirst position; and adjusting the depth of field on the basis of thedrag operation while changing the aperture value of the diaphragm-plateicon.
 9. A non-transitory computer-readable medium having embodiedthereon a program, which when executed by a computer causes the computerto execute a method, the method comprising: obtaining a first touchinput to a first position on a touch-panel display; setting a focusposition of a captured image on a basis of a distance information of thecaptured image corresponding to the first position; controlling, on abasis of the focus position, the touch-panel display to display arefocus image based on a depth of field corresponding to the focusposition and display a diaphragm-plate icon at the first position overthe refocus image, wherein the diaphragm-plate icon has a shaperepresenting a diaphragm-plate of a camera and indicates an aperturevalue corresponding to the depth of field; obtaining a second touchinput to a second position on the touch-panel display, wherein thesecond touch input is a drag operation and the second position isdifferent from the first position; and adjusting the depth of field onthe basis of the drag operation while changing the aperture value of thediaphragm-plate icon.